Methods and Systems for Demosaicing

ABSTRACT

Aspects of the present invention are related to systems and methods for image demosaicing.

FIELD OF THE INVENTION

Embodiments of the present invention relate generally to the field ofimage processing, and more specifically to spatial-domain-based methodsand systems for demosaicing.

BACKGROUND

Typically, a full-color image consists of three color components perpixel. To capture three color components at each pixel, someimage-acquisition systems may comprise three separate sensor arrays,each with a different spectral sensitivity, and a method for splittingand projecting the light entering the acquisition system onto eachspectral sensor. Other full-color image-acquisition systems may comprisea stacked-photodiode-based sensor at each pixel, wherein three colorcomponents at a pixel may be separated by the wavelength-dependentpenetration depth of the incident light within the stacked-sensor pixel.Alternatives to full-color image acquisition may compriseless-than-full-color data acquisition at each pixel and full-colorreconstruction using interpolation referred to as demosaicing.

SUMMARY

Some embodiments of the present invention may comprise methods andsystems for spatial-domain-based image demosaicing of mixed-pixel-typedata.

In some of these embodiments, a first color component may beinterpolated in a first direction and a second direction. A gradientmeasure in each direction may be determined, and a preferredinterpolation direction may be determined based on the gradientmeasures. Second-color-component and third-color-component data may beinterpolated using the interpolated first-color-component data in thepreferred interpolation direction. In some embodiments, thesecond-color-component interpolation and the third-color-componentinterpolation may be in the preferred interpolation direction.

In alternative embodiments of the present invention, a gradient measuremay be computed in a first direction and a second direction for a firstcolor component. A preferred interpolation direction may be determinedbased on the gradient measures. First-color-component data may beinterpolated in the preferred interpolation direction, andsecond-color-component and third-color-component data may beinterpolated using the interpolated first-color-component data. In someembodiments, the second-color-component interpolation and thethird-color-component interpolation may be in the preferredinterpolation direction.

In alternative embodiments of the present invention, a first colorcomponent may be interpolated in multiple directions. A gradient measurein each direction may be determined, and a preferred interpolationdirection may be determined based on the gradient measures.Second-color-component and third-color-component data may beinterpolated using the interpolated first-color-component data in thepreferred interpolation direction. In some embodiments, thesecond-color-component interpolation and the third-color-componentinterpolation may be in the preferred interpolation direction

Some embodiments of the present invention may comprise median filteringafter full-color image reconstruction.

Some embodiments of the present invention may comprise refinement ofhigh-frequency content after full-color image reconstruction.

Some embodiments of the present invention may comprise median filteringand refinement of high-frequency content after full-color imagereconstruction.

The foregoing and other objectives, features, and advantages of theinvention will be more readily understood upon consideration of thefollowing detailed description of the invention taken in conjunctionwith the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL DRAWINGS

FIG. 1A is a picture illustrating a Bayer CFA (prior art);

FIG. 1B is a picture illustrating a composite-filter CMYG CFA (priorart);

FIG. 2A is a picture illustrating an exemplary mixed-pixel-typearrangement;

FIG. 2B is a picture illustrating an exemplary mixed-pixel-type CFA;

FIG. 3 is a chart showing exemplary embodiments of the present inventioncomprising interpolation of first-color-component data in a firstdirection and a second direction and determination of a preferredinterpolation direction based on directional gradients;

FIG. 4 is a chart showing exemplary embodiments of the present inventioncomprising selecting a first color channel which may be used ininterpolation of a second color channel based on channel correlations;

FIG. 5 is a chart showing exemplary embodiments of the present inventioncomprising determination of a preferred interpolation direction based ondirectional gradients;

FIG. 6 is a chart showing exemplary embodiments of the present inventioncomprising high-frequency content refinement of interpolated colorchannels;

FIG. 7 is a chart showing exemplary embodiments of the present inventioncomprising median filtering of reconstructed full-color data;

FIG. 8 is a chart showing exemplary embodiments of the present inventioncomprising high-frequency content refinement and median filtering;

FIG. 9 is a chart showing exemplary embodiments of the present inventioncomprising determination of a preferred interpolation direction based oncomputed first-color-channel gradients;

FIG. 10 is a chart showing exemplary embodiments of the presentinvention comprising high-frequency content refinement;

FIG. 11 is a chart showing exemplary embodiments of the presentinvention comprising median filtering;

FIG. 12 is chart showing exemplary embodiments of the present inventioncomprising high-frequency content refinement and median filtering; and

FIG. 13 is a chart showing exemplary embodiments of the presentinvention comprising interpolation of first-color-component data inmultiple directions and determination of a preferred interpolationdirection based on directional gradients in the multiple directions.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Embodiments of the present invention will be best understood byreference to the drawings, wherein like parts are designated by likenumerals throughout. The figures listed above are expressly incorporatedas part of this detailed description.

It will be readily understood that the components of the presentinvention, as generally described and illustrated in the figures herein,could be arranged and designed in a wide variety of differentconfigurations. Thus, the following more detailed description of theembodiments of the methods and systems of the present invention is notintended to limit the scope of the invention but it is merelyrepresentative of the presently preferred embodiments of the invention.

Elements of embodiments of the present invention may be embodied inhardware, firmware and/or software. While exemplary embodiments revealedherein may only describe one of these forms, it is to be understood thatone skilled in the art would be able to effectuate these elements in anyof these forms while resting within the scope of the present invention.

Typically, a full-color image consists of three color components perpixel. To capture three color components at each pixel, someimage-acquisition systems may comprise three separate sensor arrays,each with a different spectral sensitivity, and a method for splittingand projecting the light entering the acquisition system onto eachspectral sensor. Other full-color image-acquisition systems may comprisea stacked-photodiode-based sensor, wherein three color components at apixel may be separated by the wavelength-dependent penetration depth ofthe incident light within the stacked-sensor pixel.

Alternatives to full-color image acquisition may compriseless-than-full-color data acquisition at each pixel and full-colorreconstruction using interpolation. A standard image-acquisition systemmay comprise a 2-dimensional (2D) sensor array and a color filter array(CFA). The colors of a scene may be captured using the single sensorarray, wherein a particular color channel may be detected at each pixelin accordance with the CFA. FIG. 1A shows the well-known Bayer CFA 10consisting of a tiling using a 2×2 cell of two green (G) filters, onered (R) filter and one blue (B) filter on a rectangular grid.Alternative color filter arrays may comprise different spatialarrangement, filter absorption spectra, number of filters or pixelshape. FIG. 1B depicts one alternative CFA, a composite-filter CMYG(cyan-magenta-yellow-green) CFA 15. Each site in these CFAs correspondsto a single photo-sensor.

Color components which are not measured at a pixel may be interpolatedusing a technique which may be referred to as demosaicing (also,demosaicking). Demosaicing methods may be specific to CFA properties,which may include spatial arrangement, filter absorption spectra, numberof filters, pixel shape and other CFA properties. Some demosaicingtechniques may introduce artifacts and may be computationally expensive.

Alternative image-acquisition systems, wherein less-than-full-color datais captured at each pixel, may comprise sensors whereby multiple colorcomponents may be measured at some pixels and single color componentsmay be measured at other pixels. An image-acquisition system as such maybe referred to as a mixed-pixel-type image-acquisition system. Some ofthese image-acquisition systems may comprise stacked-photodiode-basedsensors at some pixels, thereby acquiring multiple color components atthese pixels, and single a photo-sensor covered by a particular colorfilter at other pixels, thereby acquiring a single color component atthese other pixels.

An exemplary arrangement 20 of this type is depicted in FIG. 2A. Pixelslabeled “G” (for example 22) may correspond to a standard pixel coveredwith a green filter. Pixels labeled “R|B” (for example 24) maycorrespond to a pixel covered with a magenta filter which passes thelonger-wavelength (reddish) and shorter-wavelength (bluish) light whichmay be separated by the wavelength-dependent penetration depth withinthese stacked-sensor pixels. This arrangement may be denoted [G, R|B].In this arrangement, full-color information is sensed at each two,horizontally or vertically neighboring, pixels: a standard pixel coveredwith a green filter, whereby green spectral content may be sensed; and astacked-sensor pixel covered with a magenta filter, whereby red spectralcontent and blue spectral content may be sensed. FIG. 2B depicts thetwo-filter CFA 25 comprising standard pixels (for example 27) with agreen filter and stacked two-color pixels covered with a magenta filter(for example 29). Alternative arrangements may include [R, B|G], [B,R|G], [G, G|R|B], [G, G, G, R|B], [G, R|B, R|B, G] and otherarrangements comprising two different pixel types.

A sensor, wherein full color is detected with two pixels, may bereferred to as a “2 Pixels Full Color” (“2PFC”) sensor and the CFA maybe composed of two different types of pixels and may use only two colorfilters. With 2PFC, and other mixed-pixel-type sensors, typicallyrequired digital processing steps may be similar to those used with astandard sensor. Exemplary digital processing which may be similarincludes white balancing, tone mapping and color correction. However,standard-sensor demosaicing may not apply to mixed-pixel-type sensor.Methods and systems for demosaicing mixed-pixel-type data may bedesirable.

Other alternative image-acquisition systems, whereinless-than-full-color data is captured at each pixel, may comprise sensorarrangements whereby multiple color components may be obtained at eachpixel. An image-acquisition system as such may be referred to as astacked-pixel-type image-acquisition system. Some of theseimage-acquisition systems may comprise tilings of stacked sensors, forexample [G|R, G|B], [B|R, B|G] and other stacked-sensor tilings. Otherof these image-acquisition systems may comprise two separate sensorarrays and a method for splitting and projecting the light entering theacquisition system onto each spectral sensor, whereby one of the sensorarrays may sense one color component (for example, green), and the othersensor array may be overlaid with a CFA comprising alternating filtersassociated with the remaining color components (for example, red andgreen filters). Still other of these image-acquisition systems maycomprise a single sensor array used in conjunction with optical elementsthat allow sensing of multiple wavelength bands at each photosite. Anexemplary design of this type is disclosed in U.S. Pat. No. 7,138,663,entitled “Color separation device of solid-state image sensor.” As formixed-pixel-type image-acquisition systems, standard-sensor demosaicingmay not apply to stacked-pixel-type image-acquisition systems. Methodsand systems for demosaicing mixed-pixel-type data and stacked-pixel-typedata may be desirable.

For the purposes of describing embodiments of the present invention,image data acquired from a particular sensor arrangement may be denotedf_([ ]), wherein the subscript, [ ], may indicate the sensorarrangement. For example, f_([G,R|B]) may indicate image data acquiredfrom a mixed-pixel-type sensor arrangement comprisingsingle-color-component-data green pixels andmultiple-color-component-data stacked red/blue pixels. As anotherexample, image data acquired from a stacked-pixel-type sensorarrangement [G|R, G|B] may be denoted f_([G|R,G|B]). Color-componentchannels or combinations of color-component channels associated withsensed data may be denoted f_([ ]) ^(c1) and f_([ ]) ^(c1c2),respectively.

Generally, demosaicing algorithms may be separated into two classes:spatial-domain-based approaches and frequency-domain-based approaches.Some embodiments of the present invention comprise methods and systemsfor spatial-domain-based demosaicing.

Some embodiments of the present invention may be described in relationto FIG. 3. In these embodiments, data associated with a first colorcomponent may be interpolated 30 in a first direction. The dataassociated with the first color component also may be interpolated 32 ina second direction. In some embodiments, first-color-component datainterpolation 30, 32 may comprise filtering of first-color-componentdata and data associated with a second color component according to:

Ĉ ₁ ^(d) =h ₁ ^(d) *C ₁ +h ₂ ^(d) *C ₂,

where * denotes convolution, Ĉ₁ ^(d) denotes the first-color-componentinterpolated data in a direction denoted d, C₁ and C₂ denotefirst-color-component data and second-color-component data,respectively, and h₁ ^(d) and h₂ ^(d) denote filter kernels associatedwith direction d and the first and second color components,respectively. In some exemplary embodiments of the present invention,the first color component may be associated with green spectral contentand the second color component may be associated with red spectralcontent. In alternative exemplary embodiments of the present invention,the first color component may be associated with green spectral contentand the second color component may be associated with blue spectralcontent. In some embodiments of the present invention, the firstdirection may be the horizontal direction with respect to the imagecoordinates, and the second direction may be the vertical direction withrespect to the image coordinates. In these embodiments, thefirst-color-component data may be interpolated in the horizontaldirection and the vertical direction according to:

Ĉ ₁ ^(H) =h ₁ ^(H) *C ₁ +h ₂ ^(H) *C ₂

and

Ĉ ₁ ^(V) =h ₁ ^(V) *C ₁ +h ₂ ^(V) *C ₂,

respectively.

In alternative embodiments, the first-color-component data interpolation30, 32 may comprise filtering of first-color-component data,second-color-component data and third-color-component data according to:

Ĉ ₁ ^(d) =h ₁ ^(d) *C ₁ +h ₂ ^(d) *C ₂ +h ₃ ^(d) *C ₃,

where Ĉ₁ ^(d) denotes the first-color-component interpolated data indirection d, C₁, C₂ and C₃ denote first-color-component data,second-color-component data and third-color-component data respectively,and h₁ ^(d), h₂ ^(d) and h₃ ^(d) denote filter kernels associated withdirection d and the first, second and third color components,respectively. In some embodiments of the present invention, in order toprevent mean color shifts, the second-color-component filter and thethird-color-component filter may be jointly designed and may have zeroDC response. In some embodiments of the present invention, the firstcolor component may be associated with green spectral content, thesecond color component may be associated with red spectral content andthe third color component may be associated with blue spectral content.In some embodiments of the present invention, the first direction may bethe horizontal direction with respect to the image coordinates, and thesecond direction may be the vertical direction with respect to the imagecoordinates. In these embodiments, the first-color-component data may beinterpolated in the horizontal direction and the vertical directionaccording to:

Ĉ ₁ ^(H) =h ₁ ^(H) *C ₁ +h ₂ ^(H) *C ₂ +h ₃ ^(H) *C ₃

and

Ĉ ₁ ^(V) =h ₁ ^(V) *C ₁ +h ₂ ^(V) *C ₂ +h ₃ ^(V) *C ₃,

respectively.

A preferred interpolation direction may be determined 34, andreconstructed data associated with the first color component may beselected 36 based on the preferred direction. Second-color-componentdata may be interpolated 38 using the reconstructedfirst-color-component data, and third-color-component data may beinterpolated 40 using the reconstructed first-color-component data. Insome embodiments of the present invention, the first direction may bethe horizontal direction with respect to image coordinates, and thesecond direction may be the vertical direction with respect to the imagecoordinates. Exemplary interpolation methods comprise bilinearinterpolation, linear interpolation, spline interpolation, cubicinterpolation, cosine interpolation, Hermite interpolation, polynomialinterpolation and other interpolation methods known in the art.

In some embodiments of the present invention described in relation toFIG. 4, the correlation between the first-color-component data and thesecond-color-component data and the correlation between thefirst-color-component data and the third-color-component data may beused to determine which of the second-color-component data and thethird-color-component data may be used in interpolation 30, 32 of thefirst-color-component data in the first and the second directions. Inthese embodiments, the inter-channel correlation between thefirst-color-component data and the second-color-component data, whichmay be denoted r_(c1c2) (labeled “corr(c1,c2)” on FIG. 4), may bemeasured 42, and the inter-channel correlation between thefirst-color-component data and the third-color-component data, which maybe denoted r_(c1c3) (labeled “corr(c1,c3)” on FIG. 4), may be measured44. The relationship between r_(c1c2), and r_(c1c3) may be examined 46,and if r_(c1c2)>r_(c1c3) 47, then the second-color-component data may beused 48 to interpolate 30, 32 the first-color-component data in the twodirections. If r_(c1c2)≦r_(c1c3) 49, then the third-color-component datamay be used 50 to interpolate 30, 32 the first-color-component data inthe two directions. In alternative embodiments of the present invention,the data associated with one of the second color component or the thirdcolor component may be predetermined as the color-component data to useto interpolate the first-color-component data.

A preferred interpolation direction may be determined 34, in someembodiments of the present invention, according to FIG. 5. In theseembodiments, a first color difference may be calculated 60 in the firstdirection using the first-color-component data interpolated in the firstdirection and one of the second-color-component data and thethird-color-component data. A second color difference may be calculated62 in the second direction using the first-color-component datainterpolated in the second direction and one of thesecond-color-component data and the third-color-component data. A colordifference in a direction denoted d, may be determined 62, 64 accordingto:

C _(i,j) ^(d) =C _(k)(i,j)−Ĉ ₁ ^(d)(i,j),

where C_(k)(i,j) denotes one of the second color component and the thirdcolor component.

A gradient in the first direction may be calculated 64, and a gradientin the second direction may be calculated 66 using the respective colordifferences. In some embodiments of the present invention, a gradient ina direction d, may be determined 64, 66 according to:

D _(i,j) ^(d) =|C _(i,j) ^(d) −C _(d(i,j)) ^(d)|,

where d(i,j) denotes a directional displacement relative to i,j. In someembodiments of the present invention, the first direction may be thehorizontal direction with respect to image coordinates, and the seconddirection may be the vertical direction with respect to the imagecoordinates. In these embodiments, a horizontal gradient and a verticalgradient may be determined according to:

D _(i,j) ^(H) =|C _(i,j) ^(H) −C _(dH(i,j)) ^(H)|

and

D _(i,j) ^(V) =|C _(i,j) ^(V) −C _(dV(i,j)) ^(V)|.

In some embodiments, the horizontal displacement may be such that:D_(i,j) ^(H)=|C_(i,j) ^(H)−C_(i,j+2) ^(H)|, and the verticaldisplacement may be such that: D_(i,j) ^(V)=|C_(i,j) ^(V)−C_(i+2,j)^(V)|. In alternative embodiments, a directional gradient may becalculated 64, 66 over larger windows according to:

$D_{i,j}^{d} = {\sum\limits_{J = {{- K}\; 1}}^{K\; 1}{\sum\limits_{I = {{- K}\; 2}}^{K\; 2}{g_{I,J}^{d}C_{{i + I},{j + J}}^{d}}}}$

for a direction d, and

$D_{i,j}^{H} = {\sum\limits_{J = {{- K}\; 1}}^{K\; 1}{\sum\limits_{I = {{- K}\; 2}}^{K\; 2}{g_{I,J}^{H}C_{{i + I},{j + J}}^{H}}}}$and${D_{i,j}^{V} = {\sum\limits_{J = {{- K}\; 1}}^{K\; 1}{\sum\limits_{I = {{- K}\; 2}}^{K\; 2}{g_{I,J}^{V}C_{{i + I},{j + J}}^{V}}}}},$

for the horizontal direction and vertical direction, respectively. Theweights g_(I,J) ^(d), g_(I,J) ^(H) and g_(I,J) ^(V) may determine whichneighboring pixels may contribute to the directional gradient and withwhat weight they may contribute. In yet alternative embodiments, adirectional gradient may be based on three-channel data within a windowaccording to:

$D_{i,j}^{d} = {{\sum\limits_{J = {{- K}\; 1}}^{K\; 1}{\sum\limits_{I = {{- K}\; 2}}^{K\; 2}{{g_{2}^{d}\left( {I, J} \right)} {C_{2}\left( \begin{matrix}{{i + I},} \\{j + J}\end{matrix} \right)}}}} + {{g_{1}^{d}\left( {I, J} \right)} {{\hat{C}}_{1}^{d}\left( \begin{matrix}{{i + I},} \\{j + J}\end{matrix} \right)} {g_{3}^{d}\left( {I, J} \right)} {C_{3}\left( \begin{matrix}{{i + I},} \\{j + J}\end{matrix} \right)}}}$

where g₁ ^(d), g₂ ^(d) and g₃ ^(d) may determine which neighboringpixels, for each color component, contribute to the gradient and withwhat weight they may contribute. Where there is no color-component dataavailable, the corresponding weights may be zero.

A first-direction classification variable may be calculated 68 in thefirst direction, and a second-direction classification variable may becalculated 70 in the second direction. In some embodiments, for aparticular direction, a classification variable value at a location maybe the weighted sum of the directional-gradient values in a neighborhoodproximate to the location.

The first-direction classification variable and the second-directionclassification variable may be examined 72, and when the first-directionclassification variable and the second-direction classification variablemeet 73 a first criterion, the first direction may be the preferredinterpolation direction 74 at the location. When the first-directionclassification variable and the second-direction classification variabledo not meet 75 the first criterion, then the first-directionclassification variable and the second-direction classification variablemay be examined 76, and when the first-direction classification variableand the second-direction classification variable meet 77 a secondcriterion, the second direction may be the preferred interpolationdirection 78 at the location. When the classification variables do notmeet the first criterion and the second criterion 79, then there may beno preferred interpolation direction 80.

Exemplary embodiments of the present invention described in relation toFIG. 3, FIG. 4 and FIG. 5 may be illustrated for mixed-pixel-type datasensed in accordance with a [G, R|B] sensor arrangement. At alternatingpixels, green data or red and blue data is available for this sensorarrangement. In some exemplary embodiments of the present invention, thered data channel may be used to interpolate 30, 32 the missing greenchannel values in the horizontal direction and the vertical directionaccording to the two 5-tap FIR filters, respectively:

${\hat{G}}_{i,j}^{H} = {{\frac{1}{2}\left( {G_{1,{j + 1}} + G_{i,{j - 1}}} \right)} + {\frac{1}{4}\left( {{2\; R_{i,j}} - R_{i,{j + 2}} - R_{i,{j - 2}}} \right)}}$and${\hat{G}}_{i,j}^{V} = {{\frac{1}{2}\left( {G_{{1 + 1},j} + G_{{i - 1},j}} \right)} + {\frac{1}{4}\left( {{2\; R_{i,j}} - R_{{i + 2},j} - R_{{i - 2},j}} \right)}}$

where G_(m,n) and R_(m,n) denote sensed green-channel and red-channelvalues at a location m,n, respectively, and Ĝ_(i,j) ^(V) and Ĝ_(i,j)^(H) are the vertically interpolated and horizontally interpolatedgreen-channel values, respectively, at location i,j. In alternativeexemplary embodiments of the present invention, the blue data channelmay be used to interpolate 30, 32 the missing green channel values inthe horizontal direction and the vertical direction according to the two5-tap FIR filters, respectively:

${\hat{G}}_{i,j}^{H} = {{\frac{1}{2}\left( {G_{1,{j + 1}} + G_{i,{j - 1}}} \right)} + {\frac{1}{4}\left( {{2\; B_{i,j}} - B_{i,{j + 2}} - B_{i,{j - 2}}} \right)}}$and${\hat{G}}_{i,j}^{V} = {{\frac{1}{2}\left( {G_{{1 + 1},j} + G_{{i - 1},j}} \right)} + {\frac{1}{4}\left( {{2\; B_{i,j}} - B_{{i + 2},j} - B_{{i - 2},j}} \right)}}$

where B_(m,n) denote sensed blue-channel values at a location m,n.Alternative interpolation filters may be used.

In some of these exemplary embodiments, a preferred interpolationdirection may be determined 34 by calculating 60, 62 the horizontal andvertical color differences according to:

C _(i,j) ^(H) =R _(i,j) −Ĝ _(i,j) ^(H)

and

C _(i,j) ^(V) =R _(i,j) −Ĝ _(i,j) ^(V),

respectively, at pixel locations i,j corresponding to pixels wherein reddata may be sensed and green data interpolated. In alternative of theseexemplary embodiments, a preferred interpolation direction may bedetermined 34 by calculating 60, 62 the horizontal and vertical colordifferences according to:

C _(i,j) ^(H) =B _(i,j) −Ĝ _(i,j) ^(H)

and

C _(i,j) ^(V) =B _(i,j) −Ĝ _(i,j) ^(V),

respectively.

In some embodiments of the present invention, horizontal and verticalcolor-difference gradients may be calculated 64, 66 according to:

D _(i,j) ^(H) =|C _(i,j) ^(H) −C _(i,j+2) ^(H)|

and

D _(i,j) ^(V) =|C _(i,j) ^(V) −C _(i+2,j) ^(V)|,

respectively. Alternative embodiments of the present invention maycomprise alternative directional gradient operators.

In some embodiments of the present invention, horizontal and verticalclassification variables may be evaluated 68, 70 according to:

δ_(i,j) ^(H) =D _(i−2,j−2) ^(H) +D _(i−2,j) ^(H) +D _(i−1,j−1) ^(H) +αD_(i,j−2) ^(H) +αD _(i,j) ^(H) +D _(i+1,j−1) ^(H) +D _(i+2,j−2) ^(H) +D_(i+2,j) ^(H)

and

δ_(i,j) ^(V) =D _(i−2,j−2) ^(V) +αD _(i−2,j) ^(V) +D _(i−2,j+2) ^(V) +D_(i−1,j−1) ^(V)+_(i−1,j+1) ^(V) +D _(i,j−2) ^(V) +αD _(i,j+2) ^(V),

respectively, where in some embodiments of the present invention, theweight α=3.

In some embodiments of the present invention, the preferredinterpolation direction may be determined 34 based on the classificationvariables. In these embodiments, if the horizontal classificationvariable is larger than the vertical classification variable, then avertical edge may be proximate to pixel i,j, and the preferredinterpolation direction for interpolating the green channel at i,j maybe the vertical direction. If the vertical classification variable islarger than the horizontal classification variable, then a horizontaledge may be proximate to pixel i,j, and the preferred interpolationdirection for interpolating the green channel at i,j may be thehorizontal direction. If the classification variables are substantiallyequal, then there may be no preferred interpolation direction, and thegreen channel value at i,j may be a combination of the horizontallyinterpolated value and the vertically interpolated value. In someembodiments of the present invention, the reconstructed green channeldata may be selected 36 according to:

$\begin{matrix}{{\hat{G}}_{i,j} = {\hat{G}}_{i,j}^{H}} & {\delta_{i,j}^{H} < \delta_{i,j}^{V}} \\{{\hat{G}}_{i,j} = {\hat{G}}_{i,j}^{V}} & {\delta_{i,j}^{H} > \delta_{i,j}^{V}} \\{{\hat{G}}_{i,j} = {\frac{1}{2}\left( {{\hat{G}}_{i,j}^{H} + {\hat{G}}_{i,j}^{V}} \right)}} & {{\delta_{i,j}^{H} = \delta_{i,j}^{V}},}\end{matrix}$

where Ĝ_(i,j) be the reconstructed green pixel at location i,j. Inalternative embodiments, the green channel data may be selected 36according to:

$\begin{matrix}{{\hat{G}}_{i,j} = {\hat{G}}_{i,j}^{H}} & {{\delta_{i,j}^{V} - \delta_{i,j}^{H}} > \beta} \\{{\hat{G}}_{i,j} = {\hat{G}}_{i,j}^{V}} & {{\delta_{i,j}^{H} - \delta_{i,j}^{V}} > \beta} \\{{\hat{G}}_{i,j} = {\frac{1}{2}\left( {{\hat{G}}_{i,j}^{H} + {\hat{G}}_{i,j}^{V}} \right)}} & {{{{\delta_{i,j}^{H} - \delta_{i,j}^{V}}} \leq \beta},}\end{matrix}$

where the threshold β may account for smooth regions in the image.

In some embodiments, the red and blue channels may be interpolated 38,40 using bilinear interpolation of color differences according to:

${\hat{R}}_{i,j} = {G_{i,j} + {\frac{1}{4}\begin{pmatrix}{R_{i,{j - 1}} - {\hat{G}}_{i,{j - 1}} + R_{i,{j + 1}} - {\hat{G}}_{i,{j + 1}} +} \\{R_{{i - 1},j} - {\hat{G}}_{{i - 1},j} + R_{{i + 1},j} - {\hat{G}}_{{i + 1},j}}\end{pmatrix}}}$ and${{\hat{B}}_{i,j} = {G_{i,j} + {\frac{1}{4}\begin{pmatrix}{B_{i,{j - 1}} - {\hat{G}}_{i,{j - 1}} + B_{i,{j + 1}} - {\hat{G}}_{i,{j + 1}} +} \\{B_{{i - 1},j} - {\hat{G}}_{{i - 1},j} + B_{{i + 1},j} - {\hat{G}}_{{i + 1},j}}\end{pmatrix}}}},$

respectively, where {circumflex over (R)}_(i,j) is the reconstructed redpixel at location i,j and {circumflex over (B)}_(i,j) is thereconstructed blue pixel at location i,j.

In alternative embodiments, the red and blue channels may beinterpolated 38, 40 in a direction associated with the preferredinterpolation direction. Since the horizontal and verticalclassification variables are defined at i,j locations wherein a greenpixel may require interpolation, associated horizontal and verticalclassification variables may be defined at pixel locations whereininterpolation of the red channel and the blue channel may be required.In some embodiments of the present invention, the horizontalclassification variable and the vertical classification variable atpixel i,j, wherein location i,j may be associated with red and blueinterpolation sites, may be determined according to:

${\hat{\delta}}_{i,j}^{H} = {\frac{1}{4}\left( {\delta_{i,{j - 1}}^{H} + \delta_{i,{j + 1}}^{H} + \delta_{{i - 1},j}^{H} + \delta_{{i + 1},j}^{H}} \right)}$and${{\hat{\delta}}_{i,j}^{V} = {\frac{1}{4}\left( {\delta_{i,{j - 1}}^{V} + \delta_{i,{j + 1}}^{V} + \delta_{{i - 1},j}^{V} + \delta_{{i + 1},j}^{V}} \right)}},$

respectively. In alternative embodiments, the horizontal classificationvariable and the vertical classification variable at pixel i,j, whereinlocation i,j may be associated with red and blue interpolation sites,may be determined according to:

{circumflex over (δ)}_(i,j) ^(H)=max(δ_(i,j−1) ^(H),δ_(i,j+1)^(H),δ_(i−1,j) ^(H),δ_(i+1,j) ^(H))

and

{circumflex over (δ)}_(i,j) ^(V)=max(δ_(i,j−1) ^(V),δ_(i,j+1)^(V),δ_(i−1,j) ^(V),δ_(i+1,j) ^(V)),

respectively.

In some embodiments, red and blue channel interpolation 38, 40 may beperformed according to:

$\begin{matrix}{{\hat{R}}_{i,j} = {G_{i,j} + {\frac{1}{2}\left( {R_{i,{j - 1}} - {\hat{G}}_{i,{j - 1}} + R_{i,{j + 1}} - {\hat{G}}_{i,{j + 1}}} \right)}}} & {\delta_{i,j}^{H} < \delta_{i,j}^{V}} \\{{\hat{R}}_{i,j} = {G_{i,j} + {\frac{1}{2}\left( {R_{{i - 1},j} - {\hat{G}}_{{i - 1},j} + R_{{i + 1},j} - {\hat{G}}_{{i + 1},j}} \right)}}} & {\delta_{i,j}^{H} > \delta_{i,j}^{V}} \\{{{\hat{R}}_{i,j} = {G_{i,j} + {\frac{1}{4}\begin{pmatrix}{R_{i,{j - 1}} - {\hat{G}}_{i,{j - 1}} + R_{i,{j + 1}} - {\hat{G}}_{i,{j + 1}} +} \\{R_{{i - 1},j} - {\hat{G}}_{{i - 1},j} + R_{{i + 1},j} - {\hat{G}}_{{i + 1},j}}\end{pmatrix}}}},} & {\delta_{i,j}^{H} = \delta_{i,j}^{V}}\end{matrix}$ and $\begin{matrix}{{\hat{B}}_{i,j} = {G_{i,j} + {\frac{1}{2}\left( {B_{i,{j - 1}} - {\hat{G}}_{i,{j - 1}} + B_{i,{j + 1}} - {\hat{G}}_{i,{j + 1}}} \right)}}} & {\delta_{i,j}^{H} < \delta_{i,j}^{V}} \\{{\hat{B}}_{i,j} = {G_{i,j} + {\frac{1}{2}\left( {B_{{i - 1},j} - {\hat{G}}_{{i - 1},j} + B_{{i + 1},j} - {\hat{G}}_{{i + 1},j}} \right)}}} & {\delta_{i,j}^{H} > \delta_{i,j}^{V}} \\{{{\hat{B}}_{i,j} = {G_{i,j} + {\frac{1}{4}\begin{pmatrix}{B_{i,{j - 1}} - {\hat{G}}_{i,{j - 1}} + B_{i,{j + 1}} - {\hat{G}}_{i,{j + 1}} +} \\{B_{{i - 1},j} - {\hat{G}}_{{i - 1},j} + B_{{i + 1},j} - {\hat{G}}_{{i + 1},j}}\end{pmatrix}}}},} & {{\delta_{i,j}^{H} = \delta_{i,j}^{V}},}\end{matrix}$

respectively. In alternative embodiments, red and blue channelinterpolation 38, 40 may be performed according to:

$\begin{matrix}\begin{matrix}{{\hat{R}}_{i,j} = {G_{i,j} + {\frac{1}{2}\left( {R_{i,{j - 1}} - {\hat{G}}_{i,{j - 1}} + R_{i,{j + 1}} - {\hat{G}}_{i,{j + 1}}} \right)}}} & {{\delta_{i,j}^{V} - \delta_{i,j}^{H}} > \beta} \\{{\hat{R}}_{i,j} = {G_{i,j} + {\frac{1}{2}\left( {R_{{i - 1},j} - {\hat{G}}_{{i - 1},j} + R_{{i + 1},j} - {\hat{G}}_{{i + 1},j}} \right)}}} & {{\delta_{i,j}^{H} - \delta_{i,j}^{V}} > \beta} \\{{{\hat{R}}_{i,j} = {G_{i,j} + {\frac{1}{4}\begin{pmatrix}{R_{i,{j - 1}} - {\hat{G}}_{i,{j - 1}} + R_{i,{j + 1}} - {\hat{G}}_{i,{j + 1}} +} \\{R_{{i - 1},j} - {\hat{G}}_{{i - 1},j} + R_{{i + 1},j} - {\hat{G}}_{{i + 1},j}}\end{pmatrix}}}},} & {{{\delta_{i,j}^{H} - \delta_{i,j}^{V}}} \leq \beta}\end{matrix} & \; \\{and} & \; \\\begin{matrix}{{\hat{B}}_{i,j} = {G_{i,j} + {\frac{1}{2}\left( {B_{i,{j - 1}} - {\hat{G}}_{i,{j - 1}} + B_{i,{j + 1}} - {\hat{G}}_{i,{j + 1}}} \right)}}} & {{\delta_{i,j}^{V} - \delta_{i,j}^{H}} > \beta} \\{{\hat{B}}_{i,j} = {G_{i,j} + {\frac{1}{2}\left( {B_{{i - 1},j} - {\hat{G}}_{{i - 1},j} + B_{{i + 1},j} - {\hat{G}}_{{i + 1},j}} \right)}}} & {{\delta_{i,j}^{H} - \delta_{i,j}^{V}} > \beta} \\{{{\hat{B}}_{i,j} = {G_{i,j} + {\frac{1}{4}\begin{pmatrix}{B_{i,{j - 1}} - {\hat{G}}_{i,{j - 1}} + B_{i,{j + 1}} - {\hat{G}}_{i,{j + 1}} +} \\{B_{{i - 1},j} - {\hat{G}}_{{i - 1},j} + B_{{i + 1},j} - {\hat{G}}_{{i + 1},j}}\end{pmatrix}}}},} & {{{{\delta_{i,j}^{H} - \delta_{i,j}^{V}}} \leq \beta},}\end{matrix} & \;\end{matrix}$

respectively, where the threshold β may account for smooth regions inthe image.

Some embodiments of the present invention described in relation to FIG.6 may comprise refinement of high-frequency content.High-frequency-content refinement may eliminate reconstruction artifactsin high-frequency regions. In these embodiments, a color-component valuemay be decomposed into high-frequency and low-frequency componentsaccording to:

C _(i,j) =C _(i,j) ^(low) +C _(i,j) ^(high)

where C denotes the value at i,j associated with a color component. Thehigh-frequency component of the reconstructed second-color-componentvalues may be refined 90 according to:

C2_(i,j) =C2_(i,j) ^(low) +C1_(i,j) ^(high),

and the high-frequency component of the reconstructedthird-color-component values may be refined 92 according to:

C3_(i,j) =C3_(i,j) ^(low) +C1_(i,j) ^(high).

In some embodiments of the present invention, an FIR low-pass filter,for example

${\frac{1}{3}\lbrack 111\rbrack},$

may be used to extract low-frequency content. High-frequency content maydetermined by subtracting the low-frequency content from the componentvalue. In some embodiments of the present invention, the low-pass filtermay be applied in the preferred interpolation direction to avoidcreating artifacts along edges in the image.

In some embodiments of the present invention, the reconstructed red andblue channels may be refined 90, 92 according to:

R _(i,j) =R _(i,j) ^(low) +G _(i,j) ^(high)

and

B _(i,j) =B _(i,j) ^(low) +G _(i,j) ^(high),

respectively, where R_(i,j) ^(low) and B_(i,j) ^(low) denote thelow-frequency content of the reconstructed red and blue channel data,respectively, and G_(i,j) ^(high) denotes the high-frequency content ofthe reconstructed green channel.

In some embodiments of the present invention described in relation toFIG. 7, a reconstructed full-color image may be median filtered 94 afterdemosaicing. Color differences may vary slowly, and small variations incolor may be suppressed by median filtering. Accordingly, in some ofthese embodiments, median filtering 94 may be performed on colordifferences. In some embodiments of the present invention, medianfiltering 94 may comprise a 3×3 kernel. In some embodiments of thepresent invention, the median filter 94 may be applied only to thereconstructed pixels. In alternative embodiments, the median filter 94may be applied to all pixels. In some embodiments of the presentinvention, the R channel may be computed first, followed by the Bchannel and finally the G channel. The three channels may be computedaccording to:

R = G + median  (R − G), B = G + median  (B − G) and$G = {{\frac{1}{2}\left\lbrack {R + {{median}\mspace{14mu} \left( {R - G} \right)} + B + {{median}\mspace{14mu} \left( {B - G} \right)}} \right\rbrack}.}$

In some embodiments of the present invention, the median filtering 92may be applied once. In alternative embodiments of the presentinvention, the median filtering 92 may be sequentially applied multipletimes.

Some embodiments of the present invention described in relation to FIG.8 may comprise full-color image reconstruction followed by refinement90, 92 of high-frequency content and median filtering 96.

Alternative embodiments of the present invention may be described inrelation to FIG. 9. In these embodiments, a gradient in a firstdirection may be computed 100 for missing pixel locations associatedwith a first color component. A gradient in a second direction may becomputed 102 for the missing pixel locations associated with the firstcolor component. Generally, a directional gradient at a pixel i,j, in adirection denoted d, may be based on three-channel data within a windowcontaining the pixel according to:

$D_{i,j}^{d} = {{\sum\limits_{J = {{- K}\; 1}}^{K\; 1}{\sum\limits_{I = {{- K}\; 2}}^{K\; 2}{{g_{2}^{d}\left( {I,J} \right)}{C_{2}\left( {{i + I},{j + J}} \right)}}}} + {{g_{1}^{d}\left( {I,J} \right)}{{\hat{C}}_{1}^{d}\left( {{i + I},{j + J}} \right)}{g_{3}^{d}\left( {I,J} \right)}{C_{3}\left( {{i + I},{j + J}} \right)}}}$

where g₁ ^(d), g₂ ^(d) and g₃ ^(d) may determine which neighboringpixels, for each color component, contribute to the gradient and withwhat weight they may contribute. Where there is no color-component dataavailable, the corresponding weights may be zero.

A preferred interpolation direction may be determined 104 based on thedirectional gradient values. In some embodiments of the presentinvention, the preferred interpolation direction may be determined basedon the value of the difference of the gradient values in the firstdirection and the gradient values in the second direction. Lettingδ_(i,j)=D_(i,j) ^(d1)−D_(i,j) ^(d2) and T_(δ) be a pre-determinedthreshold value, then in some embodiments of the present invention, thepreferred interpolation direction may be the first direction whenδ_(i,j)<−T_(δ) indicating an edge in the first direction, and thepreferred interpolation direction maybe the second direction whenδ_(i,j)>T_(δ) indicating an edge in the second direction. When|δ_(i,j)|≦T_(δ) there may be no preferred interpolation direction. Inalternative embodiments of the present invention, the preferredinterpolation direction may be determined according to:

${\delta_{i,j} = {{arc}\; {\tan \left( \frac{D_{i,j}^{d\; 1}}{D_{i,j}^{d\; 2}} \right)}}},$

where d1 and d2 may denote two orthogonal directions in which thegradients may be measured. In some embodiments of the present invention,the preferred interpolation direction may be the first direction d1 whenδ_(i,j) is close to zero, and may be the second direction d2 whenδ_(i,j) is close to π/2. In alternative embodiments, the preferredinterpolation direction may be orthogonal to δ_(i,j).

Missing first-color-component data may be interpolated 106 in thepreferred interpolation direction. The fully reconstructedfirst-color-component data may be used to interpolate 108 thesecond-color-component data and to interpolate 110 thethird-color-component data. In some embodiments of the presentinvention, the second-color-component interpolation 108 may be in thepreferred interpolation direction, and the third-color-componentinterpolation 110 may be in the preferred interpolation direction.

Exemplary embodiments of the present invention described in relation toFIG. 9 may be illustrated for mixed-pixel-type data sensed in accordancewith a [G, R|B] sensor arrangement. At alternating pixels, green data orred and blue data is available for this sensor arrangement. In someexemplary embodiments of the present invention, green-channel gradientsat missing green pixel locations may be computed in the horizontaldirection 100 and the vertical direction 102 according to:

${\Delta \; H_{ij}} = {{\frac{1}{2}R_{i,{j - 2}}} - R_{i,j} + {\frac{1}{2}R_{i,{j + 2}}}}$and${{\Delta \; V_{ij}} = {{\frac{1}{2}R_{{i - 2},j}} - R_{i,j} + {\frac{1}{2}R_{{i - 2},j}}}},$

respectively. In alternative embodiments, the green-channel gradients atmissing green pixel locations may be computed in the horizontaldirection 100 and the vertical direction 102 according to:

${\Delta \; H_{i,j}} = {{\frac{1}{2}B_{i,{j - 2}}} - B_{i,j} + {\frac{1}{2}B_{i,{j + 2}}}}$and${{\Delta \; V_{i,j}} = {{\frac{1}{2}B_{{i - 2},j}} - B_{i,j} + {\frac{1}{2}B_{{i - 2},j}}}},$

respectively. In some embodiments of the present invention, the channelwith the highest correlation with the green channel may be used in theinterpolation 100, 102. In some embodiments a weighted combination ofneighboring values may be used to estimate the horizontal and verticalgradients:

${\Delta \; H_{i,j}} = {{w_{i}*\left( {{\frac{1}{2}B_{i,{j - 2}}} - B_{i,j} + {\frac{1}{2}B_{i,{j + 2}}}} \right)} + {w_{2}*\left( {{\frac{1}{2}R_{i,{j - 2}}} - R_{i,j} + {\frac{1}{2}R_{i,{j + 2}}}} \right)}}$and${\Delta \; V_{i,j}} = {{w_{i}*\left( {{\frac{1}{2}B_{{i - 2},j}} - B_{i,j} + {\frac{1}{2}B_{{i + 2},j}}} \right)} + {w_{2}*{\left( {{\frac{1}{2}R_{{i - 2},j}} - R_{i,j} + {\frac{1}{2}R_{{i + 2},j}}} \right).}}}$

The gradient information may be used to select an interpolation kernelwhich may be adaptive to local edge information. In some embodiments ofthe present invention, the preferred interpolation direction may bedetermined 104 in relation to a threshold T_(δ). Definingδ_(i,j)=ΔH_(i,j)−ΔV_(i,j), then when δ_(i,j)>T_(δ), the preferredinterpolation direction may be the vertical direction. Whenδ_(i,j)<−T_(δ), then the preferred interpolation direction may be thehorizontal direction, and when |δ_(i,j)|≦T_(δ), then there may be nopreferred interpolation direction. The green channel may be interpolated106, in some embodiments of the present invention, using bilinearinterpolation in the preferred interpolation direction according to:

$\begin{matrix}{{\hat{G}}_{i,j} = {\frac{1}{2}\left( {G_{{i - 1},j} + G_{{i + 1},j}} \right)}} & {\delta_{i,j} > T_{\delta}} \\{{\hat{G}}_{i,j} = {\frac{1}{2}\left( {G_{i,{j - 1}} + G_{i,{j + 1}}} \right)}} & {\delta_{i,j} < {- T}} \\{{\hat{G}}_{i,j} = {\frac{1}{4}\left( {G_{{i - 1},j} + G_{{i + 1},j} + G_{i,{j - 1}} + G_{i,{j + 1}}} \right)}} & {{{\delta_{i,j}} \leq T},}\end{matrix}$

In alternative embodiments, a different interpolation method may be usedin the preferred interpolation direction. Exemplary interpolationmethods comprise bilinear interpolation, linear interpolation, splineinterpolation, cubic interpolation, cosine interpolation, Hermiteinterpolation, polynomial interpolation and other interpolation methodsknown in the art.

In alternative embodiments of the present invention, the preferredinterpolation direction may be determined 104 in relation to

${\delta_{i,j} = {{arc}\; {\tan \left( \frac{\Delta \; V_{i,j}}{\Delta \; H_{i,j}} \right)}}},$

wherein the preferred interpolation direction may be orthogonal to

$\delta_{i,j} = {{arc}\; {{\tan \left( \frac{\Delta \; V_{i,j}}{\Delta \; H_{i,j}} \right)}.}}$

In these embodiments, the green channel may be interpolated 106 in thepreferred interpolation direction.

The red and green channel data may be interpolated 108, 110 using colordifferences. Letting C_(i,j) ^(GR)=Ĝ_(i,j)−R_(i,j) and C_(i,j)^(GB)=Ĝ_(i,j)−B_(i,j) be the red and blue color differences,respectively, at known red and blue pixels, respectively, thereconstructed red and blue channels may be determined according to:

${\hat{R}}_{i,j} = {G_{i,j} - {\frac{1}{4}\left( {C_{{i - 1},j}^{GR} + C_{{i + 1},j}^{GR} + C_{i,{j - 1}}^{GR} + C_{i,{j + 1}}^{GR}} \right)}}$and${{\hat{B}}_{i,j} = {G_{i,j} - {\frac{1}{4}\left( {C_{{i - 1},j}^{GB} + C_{{i + 1},j}^{GB} + C_{i,{j - 1}}^{GB} + C_{i,{j + 1}}^{GB}} \right)}}},$

respectively. In alternative embodiments of the present invention, thered and blue channels may be interpolated in the preferred interpolationdirection.

Some embodiments of the present invention described in relation to FIG.10 may comprise high-frequency content refinement 112. Some embodimentsof the present invention described in relation to FIG. 11 may comprisemedian filtering 114. Some embodiments of the present inventiondescribed in relation to FIG. 12 may comprise high-frequency contentrefinement 112 and median filtering 116.

Some embodiments of the present invention may be described in relationto FIG. 13. In these embodiments, data associated with a first colorcomponent may be interpolated 120 in multiple directions. In someembodiments, first-color-component data interpolation 120 may comprisefiltering of first-color-component data and data associated with asecond color component according to:

Ĉ ₁ ^(d) =h ₁ ^(d) *C ₁ +h ₂ ^(d) *C ₂,

where Ĉ₁ ^(d) denotes the first-color-component interpolated data in adirection denoted d, C₁ and C₂ denote first-color-component data andsecond-color-component data, respectively, and h₁ ^(d) and h₂ ^(d)denote filter kernels associated with direction d and the first andsecond color components, respectively. In some exemplary embodiments ofthe present invention, the first color component may be associated withgreen spectral content and the second color component may be associatedwith red spectral content. In alternative exemplary embodiments of thepresent invention, the first color component may be associated withgreen spectral content and the second color component may be associatedwith blue spectral content.

In alternative embodiments, the first-color-component data interpolation120 may comprise filtering of first-color-component data,second-color-component data and third-color-component data according to:Ĉ ₁ ^(d) =h ₁ ^(d) *C ₁ +h ₂ ^(d) *C ₂ +h ₃ ^(d) *C ₃,

where Ĉ₁ ^(d) denotes the first-color-component interpolated data indirection d, C₁, C₂, and C₃ denote first-color-component data,second-color-component data and third-color-component data respectively,and h₁ ^(d), h₂ ^(d) and h₃ ^(d) denote filter kernels associated withdirection d and the first, second and third color components,respectively. In some embodiments of the present invention, in order toprevent mean color shifts, the second-color-component filter and thethird-color-component filter may be jointly designed and may have zeroDC response. In some embodiments of the present invention, the firstcolor component may be associated with green spectral content, thesecond color component may be associated with red spectral content andthe third color component may be associated with blue spectral content.

A first-color-component gradient measure may be calculated 122 in eachof the multiple directions in which the first-color-component wasinterpolated 120.

A preferred interpolation direction may be determined 124 based on thegradient measures, and reconstructed data associated with the firstcolor component may be selected 126 based on the preferred interpolationdirection. Second-color-component data may be interpolated 128 using thereconstructed first-color-component data, and third-color-component datamay be interpolated 130 using the reconstructed first-color-componentdata. Exemplary interpolation methods comprise bilinear interpolation,linear interpolation, spline interpolation, cubic interpolation, cosineinterpolation, Hermite interpolation, polynomial interpolation and otherinterpolation methods known in the art.

In some embodiments of the present invention, a gradient measure in adirection denoted d, may be determined using color-component differencesaccording to:

C _(i,j) ^(d) =C _(k)(i,j)−Ĉ ₁ ^(d)(i,j),

where C_(k) (i,j) denotes one of the second color component and thethird color component. In some of these embodiments, a gradient indirection d, may be determined according to:

D _(i,j) ^(d) =|C _(i,j) ^(d) −C _(d(i,j)) ^(d)|,

where d(i,j) denotes a directional displacement relative to i,j. Inalternative embodiments, a directional gradient may be calculated overlarger windows according to:

$D_{i,j}^{d} = {\sum\limits_{J = {{- K}\; 1}}^{K\; 1}{\sum\limits_{I = {{- K}\; 2}}^{K\; 2}{g_{I,J}^{d}C_{{i + I},{j + J}}^{d}}}}$

for direction d. The weights g_(I,J) ^(d), may determine whichneighboring pixels may contribute to the directional gradient and withwhat weight they may contribute. In yet alternative embodiments, adirectional gradient may be based on three-channel data within a windowaccording to:

$D_{i,j}^{d} = {{\sum\limits_{J = {{- K}\; 1}}^{K\; 1}{\sum\limits_{I = {{- K}\; 2}}^{K\; 2}{{g_{2}^{d}\left( {I,J} \right)}{C_{2}\left( {{i + I},{j + J}} \right)}}}} + {{g_{1}^{d}\left( {I,J} \right)}{{\hat{C}}_{1}^{d}\left( {{i + I},{j + J}} \right)}{g_{3}^{d}\left( {I,J} \right)}{C_{3}\left( {{i + I},{j + J}} \right)}}}$

where g₁ ^(d), g₂ ^(d) and g₃ ^(d) may determine which neighboringpixels, for each color component, contribute to the gradient and withwhat weight they may contribute. Where there is no color-component dataavailable, the corresponding weights may be zero.

In some embodiments of the present invention, the preferredinterpolation direction may be determined 124 based ondirection-classification variables calculated in association with thegradients in each interpolation direction. In some embodiments, for aparticular direction, a classification variable value at a location maybe the weighted sum of the directional-gradient values in a neighborhoodproximate to the location according to:

${\delta_{i,j}^{d} = {\sum\limits_{J = {{- K}\; 1}}^{K\; 1}{\sum\limits_{I = {{- K}\; 2}}^{K\; 2}{w_{I,J}^{d}D_{{i + I},{j + J}}^{d}}}}},$

where w_(I,J) ^(d), may determine which neighboring gradients contributeto the direction-classification variable denoted δ_(i,j) ^(d), and withwhat weight they may contribute.

In some embodiments of the present invention, the preferredinterpolation direction at a location i,j may be orthogonal to thedirection associated with

$\max\limits_{d}{\left( \delta_{i,j}^{d} \right).}$

In alternative embodiments, when

${\max\limits_{d}\left( \delta_{i,j}^{d} \right)} \leq T_{\delta}$

for a threshold T_(δ), the preferred interpolation direction may be apre-determined direction. In other alternative embodiments, thepreferred interpolation direction may be based on other criteriaassociated with the direction-classification variables.

In some embodiments of the present invention, the second-color-componentvalues and third-color-component values may be interpolated in thepreferred interpolation direction using the reconstructedfirst-color-component values interpolated in the preferred interpolationdirection. In alternative embodiments, the second-color-component valuesand third-color-component values may be interpolated in a direction, ordirections, not related to the preferred interpolation direction.

The terms and expressions which have been employed in the foregoingspecification are used therein as terms of description and not oflimitation, and there is no intention in the use of such terms andexpressions of excluding equivalence of the features shown and describedor portions thereof, it being recognized that the scope of the inventionis defined and limited only by the claims which follow.

1. A method for demosaicing an image comprising a first plurality of pixels associated with a first color component and a second color component and a second plurality of pixels associated with, at least, a third color component, said method comprising: a) interpolating, in a first direction, missing values in said third color component; b) interpolating, in a second direction, missing values in said third color component; c) interpolating missing values in said first color component using said first-direction-interpolated third-color-component values when a first criterion comprising a first gradient-strength measure associated with said first direction and a second gradient-strength measure associated with said second direction is met; and d) interpolating missing values in said first color component using said second-direction-interpolated third-color-component values when a second criterion comprising said first gradient-strength measure associated with said first direction and said second gradient-strength measure associated with said second direction is met.
 2. A method as described in claim 1, wherein said interpolating missing values in said third color component in said first direction comprises a weighted average of a first plurality of third-color-component values and one of a first plurality of first-color-component values and first plurality of second-color-component values.
 3. A method as described in claim 2, wherein said one of a first plurality of first-color-component values and first plurality of second-color-component values is selected based on a first correlation between said first color component and said third color component and a second correlation between said second color component and said third color component.
 4. A method as described in claim 1 further comprising interpolating missing values in said first color component using said first-direction-interpolated third-color-component values and said second-direction-interpolated third-color-component values when said first gradient-strength measure associated with said first direction meets a third criterion in relation to said second gradient-strength measure associated with said second direction.
 5. A method as described in claim 1, wherein: a) said second plurality of pixels is associated only with said third color component; b) said first color component is associated with blue spectral content; c) said second color component is associated with red spectral content; and d) said third color component is associated with green spectral content.
 6. A method as described in claim 1 further comprising median filtering.
 7. A method as described in claim 1 further comprising, for a first interpolated first-color-component pixel: a) determining a high-frequency portion of a third-color-component pixel associated with said first interpolated first-color-component pixel; b) determining a low-frequency portion of said first interpolated first-color-component pixel; and c) adjusting the value of said first interpolated first-color-component pixel to the sum of said high-frequency portion and said low-frequency portion.
 8. A method as described in claim 1 further comprising refining a high-frequency portion of one of said first-color-component values, said second-color-component values and said third-color-component values based on another of said first-color-component values, said second-color-component values and said third-color-component values.
 9. A method as described in claim 1 further comprising: a) interpolating missing values in said second color component using said first-direction-interpolated third-color-component values when said first gradient-strength measure associated with said first direction meets said first criterion in relation to said second gradient-strength measure associated with said second direction; and b) interpolating missing values in said second color component using said second-direction-interpolated third-color-component values when said first gradient-strength measure associated with said first direction meets said second criterion in relation to said second gradient-strength measure associated with said second direction.
 10. A method as described in claim 9 further comprising interpolating missing values in said second color component using said first-direction-interpolated third-color-component values and said second-direction-interpolated third-color-component values when said first gradient-strength measure associated with said first direction meets a third criterion in relation to said second gradient-strength measure associated with said second direction.
 11. A method as described in claim 1, wherein: a) said interpolating missing values in said first color component using said first-direction-interpolated third-color-component values comprises interpolation in said first direction; and b) Said interpolating missing values in said first color component using said second-direction-interpolated third-color-component values comprises interpolation in said second direction.
 12. A method for demosaicing an image comprising a first plurality of pixels associated with a first color component and a second color component and a second plurality of pixels associated with, at least, a third color component, said method comprising: a) calculating, in a first direction, a first gradient associated with said third color component; b) calculating, in a second direction, a second gradient associated with said third color component; c) determining a gradient measure using said first gradient and said second gradient; d) interpolating, in an interpolation direction determined by said gradient measure, missing values in said third color component; and e) interpolating missing values in said first color component using said interpolated third-color-component values.
 13. A method as described in claim 12, wherein said calculating, in a first direction, a first gradient associated with said third color component comprises a weighted average of one of a first plurality of first-color-component values and first plurality of second-color-component values.
 14. A method as described in claim 13, wherein said one of a first plurality of first-color-component values and first plurality of second-color-component values is selected based on a first correlation between said first color component and said third color component and a second correlation between said second color component and said third color component.
 15. A method as described in claim 12 further comprising interpolating, in said first direction and said second direction, missing values in said third color component when said gradient measure meets a first criterion.
 16. A method as described in claim 12, wherein: a) said second plurality of pixels is associated only with said third color component; b) said first color component is associated with blue spectral content; c) said second color component is associated with red spectral content; and d) said third color component is associated with green spectral content.
 17. A method as described in claim 12 further comprising median filtering.
 18. A method as described in claim 12 further comprising, for a first interpolated first-color-component pixel: a) determining a high-frequency portion of a third-color-component pixel associated with said first interpolated first-color-component pixel; b) determining a low-frequency portion of said first interpolated first-color-component pixel; and c) adjusting the value of said first interpolated first-color-component pixel to the sum of said high-frequency portion and said low-frequency portion.
 19. A method as described in claim 12 further comprising refining a high-frequency portion of one of said first-color-component values, said second-color-component values and said third-color-component values based on another of said first-color-component values, said second-color-component values and said third-color-component values.
 20. A method as described in claim 12 further comprising interpolating missing values in said second color component using said interpolated third-color-component values.
 21. A method as described in claim 20, wherein said interpolating missing values in said second color component using said interpolated third-color-component values comprises interpolating in said interpolation direction determined by said gradient measure.
 22. A method as described in claim 12, wherein said interpolating missing values in said first color component using said interpolated third-color-component values comprises interpolating in said interpolation direction determined by said gradient measure.
 23. A method for demosaicing an image comprising a first plurality of pixels associated with a first color component and a second color component and a second plurality of pixels associated with, at least, a third color component, said method comprising: a) interpolating, in a first direction, missing values in said third color component; b) interpolating, in a second direction, missing values in said third color component; c) calculating a first gradient measure in said first direction using said interpolated third-color-component values interpolated in said first direction; d) calculating a second gradient measure in said second direction using said interpolated third-color-component values interpolated in said second direction e) determining a preferred interpolation direction using said first gradient measure and said second gradient measure; and f) interpolating missing values in said first color component using interpolated third-color-component values associated with said preferred interpolation direction.
 24. A method as described in claim 23 further comprising: a) interpolating, in a third direction, missing values in said third color component; b) calculating a third gradient measure in said first direction using said interpolated third-color-component values interpolated in said third direction; and c) wherein said determining a preferred interpolation direction using said first gradient measure and said second gradient measure further comprises using said third gradient measure. 